Assessing the crossdisciplinarity of technology-enhanced learning with science overlay maps and diversity measures

This paper deals with the assessment of the crossdisciplinarity of technology-enhanced learning (TEL). Based on a general discussion of the concept interdisciplinarity and a summary of the discussion in the field two empirical methods from scientometrics are introduced and applied. Science overlay maps and the Rao-Stirling-Diversity index are used to analyze the TEL field with a scientometric analysis. The science overlay maps show that a wide variety of disciplines contribute to research in the field. The analysis reveals that the field has been operating on a relatively high level of crossdisciplinarity in the last 10 years compared to 6 other fields of reference. Only in 2004 a decrease in the level of crossdisciplinarity could be identified. Introduction Technology-Enhanced Learning (TEL) has always been recognized as an interdisciplinary research field rooted in several academic disciplines like educational science, psychology/cognitive science and computer science. While the topic of interdisciplinarity has been recently discussed more intensively in the field (Sutherland, Eagle, & Joubert, 2012; Carmichael, 2011; Conole, Scanlon, & Mundin, 2010) there is very little work about how the interdisciplinarity of Technology-Enhanced Learning can be empirically assessed and compared to other fields of reference. In this paper we discuss the theoretical foundations of interdisciplinarity and summarize the discussion within the TEL community. To operationalize the concept of interdisciplinarity we present methods by which interdisciplinarity can be empirically measured. We apply science-overlay maps and the Rao-Stirling diversity measurement to provide an empirical basis for the future discussion about interdisciplinary of Technology-Enhanced Learning and compare the interdisciplinarity to other exemplary fields. The paper is structured as follows. In the next section we summarize the discussion about interdisciplinarity in general and then specifically for the field of TEL and we summarize different approaches to measure interdisciplinarity. In section 3 we provide an overview about the approach we have followed. Section 4 presents our results and in the final section we discuss our findings and limitations of the approach. Theoretical Background Modes of Interdisciplinarity The ongoing debate about the contribution of science to the grand challenges of our time constantly leads to a criticism about the disciplinarity structure of science. We define a scientific discipline with Aram (2004) as “thought domains – quasi-stable, partially integrated, semi-autonomous intellectual conveniences – consisting of problems, theories, and methods of investigation”. Gibbons et al. (1997) have published a prominent contribution to the discussion about disciplines and interdisciplinarity. The authors describe two different modes of knowledge production by the sciences. Mode 1 depicts the classical way of knowledge production in scientific disciplines. Knowledge is produced without taking into account its application context and the value of the knowledge is only assessed in an academic context. This context mostly consists of a specific academic discipline. In contrast mode 2 stands for the opening of science and has the following distinct features: Context of application, transdisciplinarity, heterogeneity, reflexivity/social accountability and novel quality control (Hessels & van Lente, 2008). Knowledge in Mode 2 is always produced within the context of application: “Such knowledge is intended to be useful to someone whether in industry or government, or society more generally and this imperative is present from the beginning” (Gibbons et al., 1997). For this paper the most important feature of Mode 2 is the aspect of transdisciplinarity. In this paper we build on earlier definitions of interdisciplinarity, crossdisciplinarity and transdisciplinarity as defined by Wagner et al. (2011). Table 1: Working definition of transdisciplinarity, crossdisciplinarity and interdisciplinarity as defined by Wagner at al. (2011) Transdisciplinarity Cooperation between scientists and practitioners Crossdisciplinarity Any form of scientific cooperation between scientific disciplines without any further explication of shared methods, goals and mutual interest Interdisciplinarity Collaboration where various disciplines, keeping their own autonomy (i.e. without becoming a serving discipline), solve a given problem, which cannot be solved by one discipline alone, in a joint way Besides this analytical perspective on interdisciplinarity there is also a sociocultural perspective that analyses interdisciplinarity as a learning process itself. Lattuca (2002) argues that the concept of apprenticeship is a core concept to understand how individuals from different academic disciplines learn from each other. She presents a study that analyses interdisciplinary scholarship from a learning and apprenticeship perspective and she qualitatively reports examples of the relational, mediated, transformative and situated nature of interdisciplinarity. Wagner et al. (2011) discuss a social perspective and a cognitive perspective to interdisciplinarity. While the social perspective takes interaction in teams of researchers from different disciplines as the anchor point, the cognitive perspective understands interdisciplinarity as cognitive process of individuals integrating concepts, methods and findings from different disciplines into own concepts and publications. Aboelela et al. (2007) present a literature review about interdisciplinary research. The authors discuss several approaches that can be taken for the study of interdisciplinary research: • In an interaction-oriented approach interdisciplinarity is analysed in terms of the ongoing cooperation between members of different scientific communities. • In a communication-oriented approach interdisciplinarity is analysed on the basis of communication happening between members of different scientific fields. • In a conceptual approach interdisciplinarity is seen as how concepts, ideas and models are integrated in the inquiry phase for solving a joint problem. • In an outcome-oriented approach the results of the cooperation are analysed. The outcome can be a joint product, prototype but also joint scientific publications. In the next section we summarize the discussion about interdisciplinarity in the field of Technology-Enhanced Learning. Interdisciplinarity in the field of Technology-Enhanced Learning and the Learning Sciences TEL is defined as an interdisciplinary field of research to which a number of disciplines contribute, namely: cognitive science, educational psychology, computer science, anthropology, sociology, information sciences, neurosciences, education, design studies, instructional design and others. While in most definitions TEL is more focused around the design of new technologies and their effect on learning and educational processes, the field of Learning Sciences is more concerned with forming theories about learning in relation to the technologies and tools used. Kolodner (2004) defines the Learning Sciences as a field dealing with the harvesting of “theories of active, constructivist, and participatory learning to design software and learning environments and ways of educating that promote deep and lasting learning”. Sawyer (2008) formulates the goal of the Learning Sciences as follows: “The goal of the learning sciences is to better understand the cognitive and social processes that result in the most effective learning, and to use this knowledge to redesign classrooms and other learning environments so that people learn more deeply and more effectively”. According to the author the field was established in the late 1980s based on the recognition that new scientific methods are needed that go beyond their own research field. We define technology-enhanced learning in a broad sense as any learning activity supported by digital technologies. Kirby, Hoadley, & Carr-Chellman (2005) have studied the relationship of the Instructional System Design (ISD) field and the Learning Sciences with a citation analysis. As a result of the analysis the authors state that the two fields are rather disconnected although they both focus on the use and development of technologies for learning. Hence, the dataset used for this analysis was rather small and it is questionable if we can identify the relation between two scientific disciplines or their level of interdisciplinarity based on such limited data. Plowman, Misailidou, & Laurillard (2007) have analysed 119 proposals submitted to the ESRC’s Teaching and Learning Research Programme (TLRP) in 2006. One of the core questions to be answered was the existence of “authentic interdisciplinary integration and innovation through applications from interdisciplinary teams that would bring together understandings of both learning and technology and draw upon perspectives from many disciplines”. In this study the profiles of applicants for grants in Technology-Enhanced Learning have been analyzed according to their departmental affiliation and disciplinary area. A multitude of disciplines could be identified in this study, but it is stated in the report that some proposals were not able to make the important step from multidisciplinary to truly integrated interdisciplinarity. Bransford et al. (2005) approach the topic of interdisciplinarity from the position that the learning sciences are in a decade of synergy between communities focusing on three research topics, namely cognitive neuroscience, technology-enhanced learning and nonformal learning support. They argue that these research fields have operated to date relatively independently from each other without further specifying on which empirical basis this finding about the current state of interdisciplinarity of the field is based. Jacobs & Ip (2005) have conducted an analysis of journal articles about computerassisted learning in the domain of health sciences as an example of existing “research islands” within the field of computer-assisted learning. The analysis reveals a citation bias towards the journal discipline in which the paper was published. Conole, Scanlon & Mundin (Conole et al., 2010) have analysed interdisciplinary challenges of the field. According to the authors interdisciplinarity is nowadays a requirement effecting decisions on a policy level (e.g. the EPSRC/ESRC research program on Technology Enhanced Learning in the UK), but interdisciplinary practices have not been studied sufficiently. They present a small-scale study with 18 participants to analyse the interdisciplinary background and cooperation in different research projects. The analysis of interviews showed that researchers have a variety of different academic backgrounds and thus a mixture of methods and approaches is the standard situation rather than an exception. Interestingly, some participants stress that not only their concept of Technology-Enhanced Learning is interdisciplinary but they define the educational sciences as interdisciplinary as a whole. In addition participants identified a number of influential theories and approaches, but more importantly they mentioned diversity as direct result from interdisciplinary working contexts as the most influential aspect of their work. Carmichael (Carmichael, 2011) approaches the interdisciplinarity of the field from a “culture of inquiry” perspective. This approach is more focused on the actual joint practices and discourses established during work of individuals from different scientific communities in a national project in the UK. He follows an interaction-oriented approach to study in detail the narratives and points-of-focus emerging in the project. The analysis reveals that the participants are well aware of the interdisciplinary nature of their work and that three kinds of components are needed to enable interdisciplinarity in the TEL field: awareness about different conceptualizations, discussion spaces and new ways of writing research to enable a discourse in the writing process. While the work presented in this section makes important conceptual and theoretical contributions for the discussion about the interdisciplinarity of TEL and the Learning Sciences there are three striking problems with this work: First there is mostly no operational and measurable definition of interdisciplinarity provided. Second the data basis is small. This leads to a limited representativeness of the findings. A third issue relates to the scalability of the methods chosen. Most researchers approach the study of crossdisciplinarity from a qualitative perspective. This has clearly limitations in terms of scalability. In the next section we present a methodology to measure the degree of interdisciplinarity quantitatively based on a scientometric approach that addresses the problem of scalability and representativeness. Since we focus in this paper on an outcome-oriented approach that does not take into account the quality of cooperation as it is important for the definition of interdisciplinarity we will use the term “crossdisciplinarity” as an overarching concept. Measuring Crossdisciplinarity Quantitatively Wagner et al. (2011) discuss two perspectives for measuring interdisciplinary research: The authors compare a structuralist approach with an approach that uses spatial distance as an assessment instrument. The structuralist approach focuses on unveiling and using the underlying social and cognitive structure of scientific disciplines as level of analysis. Haythornthwaite (2006) follows a structuralist approach based on the analysis of social networks and interaction types in these networks. Nine different types of learning processes happening in three interdisciplinary teams are analyzed from a reciprocal perspective (learning received/provided). The ratio between these two perspectives and the attention of an individual towards exchange and interaction inside and outside of the own core network is combined to assess the level of interdisciplinarity. An alternative approach to measure interdisciplinary scientific research (IDR) uses bibliographic analysis also called bibliometrics or scientometrics. The most prominent platforms for conducting a bibliographic analysis are the Web of Science (WoS) by Thompson Reuters, Scopus by Elsevier and GoogleScholar by Google. Although some authors argue for the use of GoogleScholar to overcome limitations of WoS (Van Aalst, 2010), the Web of Science is still regarded as the most reliable platform for all kinds of bibliographic analysis. Two core aspects influence the reliability of the citation analysis, namely the aggregation level (author, organisation, field etc.) and the level of knowledge integration being understood as how much researchers have integrated methods or findings from other disciplines into their own concepts and publications. Using the affiliation of co-authors to assess interdisciplinarity is a problematic and not valid measurement method according to Wagner et al. (2011) since this information source is not sufficiently reliable and the research field cannot be deducted from affiliation information. The most useful indicator for a citation analysis to measure the interdisciplinarity is the percentage of citations outside of the discipline of the citing paper. The critical point here is to identify what counts as a discipline and what not. Another complicating factor is that disciplines exist in different sizes. Therefore indicators have been developed that take into account relative measures instead of total measures. One of the most developed approaches that takes into account the dynamic structure of journals belonging to scientific disciplines and the citations inside and outside the discipline has been proposed by Leydesdorff (2007). This spatial approach describes a landscape or space in which science operates from the viewpoint of a single entity (topic, journal, author etc). The spatial distance in such a space is used and combined with new approaches to visualize these data. Often factor analysis is used to identify emergent structures of the dataset. The approach presented here is based on the work of Porter & Rafols (2009) who have introduced the concept of an integration score to operationalize and measure the level of interdisciplinarity of a research group, an institution or even a research field. The concept is based on the idea that the level of interdisciplinarity can be assessed by an analysis of three different aspects: • the variety of the field (number of disciplines cited), • the balance of the field (distribution of citations between fields) and the • disparity of disciplines cited (how similar are these disciplines). This combination of perspectives to measure interdisciplinary research is aligned with earlier approaches to measure diversity by Rao (1982) and Stirling (2007). Hence, the presented integration score is a special form of the Rao-Stirling Index of Diversity. This index can be understood as “a Simpson diversity in which the products of proportions of categories are weighted by distance/similarity” (Rafols & Meyer, 2010). The basis for the method applied in this paper is a global map of science constructed with the help of the subject categories (SC) in the Web of Science (Loet Leydesdorff & Rafols, 2009). Each journal in the Essential Science Indicators Database in the WoS (12 845 journals) has been assigned to one of 22 broad fields of research. In addition there are ca. 172 subject categories that are more fine-grained than the 22 fields. Rafols, Porter, & Leydesdorff (2010) have constructed a matrix and global map of science with the analysis of citing Subject Categories and cited subject categories in the Science Citation Index (SCI) and the Social Science Citation Index (SSCI) (in total 221 SCs) resulting in 18 macrodisciplines and their relations after a factor analysis. The authors report that the resulting maps have been proven to be stable compared to other approaches to build a global map of science (Leydesdorff, Carley, & Rafols, I., 2012). This global map of science is now combined and overlayed with a local map resulting from a specific search approach in the WoS databases. This method is used to locate a specific organization, individual or topic on the global map of science. Method For the analysis of the interdisciplinarity of the field of Technology-Enhanced Learning we have chosen the approach as presented above. This approach is using scientific publications as the basis of analysis. The level of interdisciplinarity and the evolution over time is analyzed with the help of the subject categories of the “Web of Science” by Thompson Reuters (WoS). In a first step we have conducted a search in the WoS version 5.6 in July 2012 for the period between 2002 and 2011 in the SSCI (Social Science Citation Index) and SCIE (Science Citation Index Expanded) databases with the following keywords: “learning sciences”, “technology-enhanced learning”, “computerbased training”, “elearning”, “e-learning”, “mobile learning”, “electronic learning environments” and “educational technology”. This query resulted in 4255 records. We have narrowed the results down to journal articles and proceedings papers and we have only included articles in English. The resulting list consists of 3490 records. This list has been manually checked to confirm the articles belong to the field. 14 results have been manually deleted from the original list so that finally 3476 records have been used for further analysis. Table 2 presents some standard bibliographic details about the dataset collected. Table 2: Dataset statistics for science overlay mapping analysis of Technology-Enhanced Learning Sum of the times cites 17999 Sum of Times Cited without self-citations 15274 Citing Articles 12292 Citing Articles without self-citations 11118 Average Citations per Item 5.18